PRE-TRAINED MODELS

Accelerating BERT Inference for Sequence Labeling via Early-Exit (2021.acl-long)

Challenge:	lack of large-scale open Japanese image-text pairs poses a significant barrier to the development of vision-language models.
Approach:	They construct large-scale Japanese image-text pairs using machine translation and pre-trained CLIP models on a Japanese dataset.
Outcome:	The results show that pre-trained models achieve competitive average scores on Japanese culture tasks compared to models of similar size.

Copied to clipboard

Xiaonan Li, Yunfan Shao, Tianxiang Sun, Hang Yan, Xipeng Qiu, Xuanjing Huang

DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization (2022.acl-short)

Challenge:	Existing early-exit mechanisms are designed for sequence-level tasks, rather than sequence labeling.
Approach:	They propose to extend sentence-level early-exit to accelerate inference of PTMs . they propose a token-level mechanism that allows partial tokens to exit early at different layers .
Outcome:	The proposed approach can save up to 66%75% inference cost with minimal performance degradation.

Copied to clipboard

Zheng Li, Zijian Wang, Ming Tan, Ramesh Nallapati, Parminder Bhatia, Andrew Arnold, Bing Xiang, Dan Roth

Cost-effective Deployment of BERT Models in Serverless Environment (2021.naacl-industry)

Challenge:	Empirical analyses show that pre-trained sequence-to-sequence models can achieve a 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts.
Approach:	They propose to distill and quantize pre-trained sequence-to-sequence models to reduce memory and latency requirements.
Outcome:	Empirical results show that the proposed model achieves 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts on multiple summarization and QA datasets.

Copied to clipboard

Marek Suppa, Katarína Benešová, Andrej Švec

ESPnet-ST: All-in-One Speech Translation Toolkit (2020.acl-demos)

Challenge:	a large upfront infrastructure investment makes machine learning models difficult to deploy . however, serverless architectures have strict limits on the size of the deployment package .
Approach:	They propose to fine-tune BERT-style models on proprietary datasets for tasks . they use knowledge distillation to obtain models that are tuned for a specific domain .
Outcome:	The proposed model deployments report acceptable latency levels and cost-effectiveness without infrastructure overhead.

Copied to clipboard

Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, Shinji Watanabe

Toward Stronger Textual Attack Detectors (2023.findings-emnlp)

Challenge:	ESPnet-ST is a new project for the quick development of speech-to-speech translation systems.
Approach:	They propose a framework for rapid development of speech-to-speech translation systems . they provide all-in-one recipes including data pre-processing, feature extraction, training, and decoding pipelines .
Outcome:	The proposed model outperforms the current state-of-the-art models on a wide range of benchmark datasets.

Copied to clipboard

Pierre Colombo, Marine Picot, Nathan Noiry, Guillaume Staerman, Pablo Piantanida

Robustness in NLP

Challenge:	despite the high performance of deep learning techniques, trained models remain vulnerable to adversarial attacks . authors present LAROUSSE, STAKEOUT and other approaches to detect adversarials . LARousSE is unsupervised, hyperparameter free and non-differentiable .
Approach:	They propose a framework to detect adversarial attacks and an extended benchmark to test them . they demonstrate that LAROUSSE outperforms previous methods and allows to identify interesting factor of detection rate variations.
Outcome:	The proposed framework outperforms existing methods and allows to identify interesting factor of detection rate variations.

RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder (2022.emnlp-main)

Copied to clipboard

Shitao Xiao, Zheng Liu, Yingxia Shao, Zhao Cao

PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation (2021.acl-short)

Challenge:	Existing methods for dense retrieval are not effective, but there are still challenges.
Approach:	They propose a retrieval oriented pre-training paradigm based on Masked Auto-Encoder (MAE) where the sentence embedding is generated from the encoder’s masked input and the original sentence is recovered based upon the sentence embedded and decoded input via mangled language modeling.
Outcome:	The proposed model significantly improves the SOTA performance on a wide range of NLP benchmarks, like BEIR and MS MARCO.

Copied to clipboard

Jing Gu, Qingyang Wu, Chongruo Wu, Weiyan Shi, Zhou Yu

Dialogue Systems & Conversational Agents

Challenge:	Existing approaches to building task-oriented dialog systems require a substantial amount of annotations and thus are labor-intensive.
Approach:	They propose a Pre-trainedRole Alternating Language model (PRAL) that is explicitly designed for task-oriented dialog tasks.
Outcome:	The proposed model outperforms or is on par with state-of-the-art models on task-oriented dialog tasks.

UER: An Open-Source Toolkit for Pre-training Models (D19-3)

Copied to clipboard

Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, Xiaoyong Du

OpenICL: An Open-Source Framework for In-context Learning (2023.acl-demo)

Challenge:	Existing work on pre-training models have shown that it is important to use a framework to deploy various pre- training models efficiently.
Approach:	They propose an assemble-on-demand pre-training toolkit that assembles pre-trained models on demand and encapsulates them with rich modules.
Outcome:	The proposed framework can reproduce state-of-the-art models or develop models that remain unexplored.

Copied to clipboard

Zhenyu Wu, Yaoxiang Wang, Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Jingjing Xu, Yu Qiao

Soloist: Building Task Bots at Scale with Transfer Learning and Machine Teaching (2021.tacl-1)

Challenge:	In-context Learning (ICL) is a new paradigm for large language model evaluation.
Approach:	They propose an open-source toolkit for ICL and LLM evaluation.
Outcome:	The proposed framework is highly flexible and flexible and can be easily combined with other tools to suit users' needs.

Copied to clipboard

Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, Jianfeng Gao

Dialogue Systems & Conversational Agents

Challenge:	Existing methods for building task-oriented dialog systems are limited to a few tasks and domains.
Approach:	They propose a method that uses transfer learning and machine teaching to build task bots at scale.
Outcome:	The proposed method outperforms existing methods on well-studied task-oriented dialog benchmarks on well studied tasks.

Enhancing Clinical BERT Embedding using a Biomedical Knowledge Base (2020.coling-main)

Copied to clipboard

Boran Hao, Henghui Zhu, Ioannis Paschalidis

Bias in Opinion Summarisation from Pre-training to Adaptation: A Case Study in Political Bias (2024.eacl-long)

Challenge:	Domain knowledge is important for building Natural Language Processing (NLP) systems for low-resource settings, such as in the clinical domain.
Approach:	They propose a joint method for adding knowledge base information from the Unified Medical Language System (UMLS) into language model pre-training for some clinical domain corpus.
Outcome:	The proposed method outperforms existing models on three clinical domain tasks with no knowledge base information.

Copied to clipboard

Nannan Huang, Haytham Fayek, Xiuzhen Zhang

Summarization

Challenge:	Existing studies have focused on extractive summarisation but limited attention has been paid to abstractive summaries.
Approach:	They propose to trace bias in abstractive summarisation models to social media opinions using different models and adaptation methods.
Outcome:	The proposed model is compared with other models and adaptation methods to summarise social media opinions using different models and adaption methods.

Suppressing Final Layer Hidden State Jumps in Transformer Pretraining (2026.findings-eacl)

Copied to clipboard

Keigo Shibata, Kazuki Yano, Ryosuke Takahashi, Jaesung Lee, Wataru Ikeda, Jun Suzuki

Learning High-Quality and General-Purpose Phrase Representations (2024.findings-eacl)

Challenge:	Existing models exhibit only slight changes in the angular distance between the input and output hidden state vectors in the middle layers .
Approach:	They propose a jump-suppressing regularizer which penalizes large hidden state displacements near the final layer during pre-training.
Outcome:	The proposed method significantly reduces hidden state jumps in the final layer and increases model capacity.

Copied to clipboard

Lihu Chen, Gael Varoquaux, Fabian Suchanek

Low-Resource Fast Text Classification Based on Intra-Class and Inter-Class Distance Calculation (2025.coling-main)

Challenge:	Pre-trained language models for phrasal embeddings are unnecessarily complex and require to be pre-tuned on a corpus with context sentences.
Approach:	They propose a framework to learn phrase representations in a context-free fashion.
Outcome:	The proposed framework generates superior phrase embeddings while requiring a smaller model size.

Copied to clipboard

Yanxu Mao, Peipei Liu, Tiehan Cui, Congying Liu, Datao You

Information Extraction & Text Mining

Challenge:	Existing methods based on neural networks and pre-trained models consume substantial memory for training and text-graph construction. Existing models require access to the test dataset during the training phase, which means that when encountering new text data, the existing model needs to be retrained.
Approach:	They propose a low-resource and fast text classification model called LFTC to address these challenges by mining regularity information within intra-class data.
Outcome:	The proposed model improves performance and processing time under limited computational and data resources on 9 publicly available datasets.

XDoc: Unified Pre-training for Cross-Format Document Understanding (2022.findings-emnlp)

Copied to clipboard

Jingye Chen, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei

As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages (2021.findings-acl)

Challenge:	Existing pre-trained models target one document format at a time, making it difficult to combine knowledge from multiple document formats.
Approach:	They propose a unified pre-trained model which deals with different document formats in a single model.
Outcome:	The proposed model achieves comparable or even better performance on a variety of downstream tasks compared with the individual pre-trained models.

Copied to clipboard

Wietse de Vries, Malvina Nissim

LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021.naacl-main)

Challenge:	Existing pre-trained language models are limited in their ability to train for English, which is a problem for many other languages.
Approach:	They propose to adapt existing generative language models to new languages by retraining lexical embeddings without tuning the Transformer layers.
Outcome:	The proposed method achieves lexical embeddings for Italian and Dutch that are aligned with the original English lexicals.

Copied to clipboard

Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, Jingjing Liu

Emergent Abilities in Reduced-Scale Generative Language Models (2024.findings-naacl)

Challenge:	Existing pre-trained models suffer from slow inference speed due to cross-modal attention in transformer architecture.
Approach:	They propose a multimodal approach that accelerates the inference time of ITR by thousands of times . they extract pre-cached feature indexes offline and employ instant dot-product matching online .
Outcome:	The proposed approach outperforms existing models that consume 1000 times magnitude of computational hours using the same features.

Copied to clipboard

Sherin Muckatira, Vijeta Deshpande, Vladislav Lialin, Anna Rumshisky

El Volumen Louder Por Favor: Code-switching in Task-oriented Semantic Parsing (2021.eacl-main)

Challenge:	Large language models can solve new tasks without task-specific fine-tuning.
Approach:	They propose to use pre-training data to pre-train 36 language models with billions of parameters to investigate whether emergent properties are tied to model size or can be demonstrated by smaller models.
Outcome:	The proposed model performs comparable to models trained on unrestricted language.

Copied to clipboard

Arash Einolghozati, Abhinav Arora, Lorena Sainz-Maza Lecanda, Anuj Kumar, Sonal Gupta

Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator (2022.findings-emnlp)

Challenge:	Code-switching (CS) is the alternation of languages within an utterance or conversation.
Approach:	They propose to use translation-and-align and augment with a generation model followed by match-and filter to improve CS generalizability of cross-lingual models when data for only one language is available.
Outcome:	The proposed models improve when only English data is available alongside zero or a few CS training instances.

Copied to clipboard

Qinyuan Cheng, Linyang Li, Guofeng Quan, Feng Gao, Xiaofeng Mou, Xipeng Qiu

Dialogue Systems & Conversational Agents

Challenge:	Task-oriented dialogue systems are drawing more attention in recent studies . current evaluation methods use annotated utterances in multi-turn dialogue sessions .
Approach:	They propose an interactive evaluation framework for task-oriented dialogue (TOD) they build a goal-oriented user simulator based on pre-trained models and use it to generate dialogues .
Outcome:	The proposed evaluation framework achieves 98% inform and success rates in the multi-woz dataset and the proposed scores measure the response quality besides the inform and successful rates.

Effectiveness of Pre-training for Few-shot Intent Classification (2021.findings-emnlp)

Copied to clipboard

Haode Zhang, Yuwei Zhang, Li-Ming Zhan, Jiaxin Chen, Guangyuan Shi, Xiao-Ming Wu, Albert Y.S. Lam

TWT: Table with Written Text for Controlled Data-to-Text Generation (2021.findings-emnlp)

Challenge:	Existing paradigms further pre-train language models such as BERT on vast amount of unlabeled corpus, but we find it highly effective and efficient to simply fine-tune BERT with roughly 1,000 labeled utterances from public datasets.
Approach:	They propose to fine-tune BERT with a small set of labeled utterances from public datasets to achieve a pre-trained model based on a set of 1,000 labeles.
Outcome:	The proposed model can outperform existing models on domains with very different semantics on novel domains.

Copied to clipboard

Tongliang Li, Lei Fang, Jian-Guang Lou, Zhoujun Li

Towards Table-to-Text Generation with Numerical Reasoning (2021.acl-long)

Challenge:	Existing methods output hallucinated text that is not faithful on TWT.
Approach:	They propose to generate text conditioned on the structured data and a prefix by leveraging pre-trained neural models.
Outcome:	The proposed approach outperforms state-of-the-art methods under automatic and human evaluation metrics.

Copied to clipboard

Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, Hiroya Takamura

MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation (2022.naacl-main)

Challenge:	Recent studies have shown improvement in generating descriptive text from structured data.
Approach:	They propose a framework for numerical table-to-text generation based on numerical reasoning . they use a pre-trained model and a copy mechanism to fine-tune the models to produce fluent text .
Outcome:	The proposed framework lacks fidelity to the table contents and is based on a pre-trained model and a copy mechanism.

Copied to clipboard

Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He, Tuo Zhao, Weizhu Chen

Query-as-context Pre-training for Dense Passage Retrieval (2023.emnlp-main)

Challenge:	Existing methods for training pre-trained language models have limited practicality due to latency requirements.
Approach:	They propose a method that uses a Mixture-of-Experts structure to increase model capacity and inference speed.
Outcome:	The proposed method outperforms existing distillation methods on natural language understanding and question answering tasks.

Copied to clipboard

Xing W, Guangyuan Ma, Wanhui Qian, Zijia Lin, Songlin Hu

Information Retrieval

Challenge:	Existing methods to improve passage retrieval performance by using context-supervised pre-training are weakly correlated.
Approach:	They propose to use query-as-context pre-training to train passage-query pairs . they evaluate the pre-trained models on large-scale passage retrieval benchmarks .
Outcome:	The proposed technique improves performance on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks.

Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words (2022.coling-1)

Copied to clipboard

Haochun Wang, Chi Liu, Nuwa Xi, Sendong Zhao, Meizhi Ju, Shiwei Zhang, Ziheng Zhang, Yefeng Zheng, Bing Qin, Ting Liu

Ethos: Rectifying Language Models in Orthogonal Parameter Space (2024.findings-naacl)

Challenge:	Pre-trained models perform poorly with limited data and rare biomedical words.
Approach:	They propose to use prompt to fine-tune pre-trained models for biomedical domain tuning with a simple approach.
Outcome:	The proposed method achieves up to 6% improvement in biomedical natural language inference task without any extra parameters or training steps using few-shot vanilla prompt settings.

Copied to clipboard

Lei Gao, Yue Niu, Tingting Tang, Salman Avestimehr, Murali Annavaram

CodeBERT: A Pre-Trained Model for Programming and Natural Languages (2020.findings-emnlp)

Challenge:	Language models (LMs) generate toxic, biased content and reveal private training records.
Approach:	They propose an efficient approach that rectifies LMs to mitigate toxicity and bias . Ethos distinguishes general beneficial and undesired knowledge when reconstructing task vectors .
Outcome:	The proposed approach mitigates toxicity and bias in outputs and avoids privacy leakage.

Copied to clipboard

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, Ming Zhou

bert2BERT: Towards Reusable Pretrained Language Models (2022.acl-long)

Challenge:	Large pre-trained models have improved performance on a variety of natural language processing tasks.
Approach:	They develop a bimodal pre-trained model for programming language (PL) and natural language (NL) it incorporates a hybrid objective function that detects replaced tokens from generators.
Outcome:	The proposed model performs better on two NL-PL applications by fine-tuning model parameters.

Copied to clipboard

Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, Qun Liu

Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification (2021.findings-acl)

Challenge:	Pre-training large language models can be expensive and wasteful.
Approach:	They propose a method which can transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and a two-stage learning method to further accelerate the pre-training.
Outcome:	The proposed method can transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and significantly improve the pre-training efficiency of the large model.

Copied to clipboard

Yijin Xiong, Yukun Feng, Hao Wu, Hidetaka Kamigaito, Manabu Okumura

Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision (2021.emnlp-main)

Challenge:	Existing methods to improve text classification performance of pre-trained models have been used to improve their performance.
Approach:	They propose a method for improving BERT's performance by using a label embedding technique while keeping almost the same computational cost.
Outcome:	The proposed method improves BERT's performance on six text classification benchmark datasets while keeping almost the same computational cost.

Copied to clipboard

Mieradilijiang Maimaiti, Yang Liu, Yuanhang Zheng, Gang Chen, Kaiyu Huang, Ji Zhang, Huanbo Luan, Maosong Sun

Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching (2022.findings-naacl)

Challenge:	Recent state-of-the-art (SOTA) effective neural network methods have been used in Chinese word segmentation (CWS) However, the robustness of the previous neural methods is limited by the large-scale annotated corpus.
Approach:	They propose a self-supervised Chinese word segmentation approach with a straightforward and effective architecture.
Outcome:	The proposed approach outperforms previous methods on 9 different CWS datasets with single criterion training and multiple criteria training and achieves better robustness.

Copied to clipboard

Kunbo Ding, Weijie Liu, Yuejian Fang, Zhe Zhao, Qi Ju, Xuefeng Yang, Rong Tian, Zhu Tao, Haoyan Liu, Han Guo, Xingyu Bai, Weiquan Mao, Yudong Li, Weigang Guo, Taiqiang Wu, Ningyuan Sun

AutoCAD: Automatically Generate Counterfactuals for Mitigating Shortcut Learning (2022.findings-emnlp)

Challenge:	Existing studies have shown that cross-lingual knowledge distillation can improve the performance of pre-trained models for cross-linguistic similarity matching tasks.
Approach:	They propose a multi-stage distillation framework for constructing a small-size but high-performance cross-lingual model using contrastive learning, bottleneck, and parameter recurrent strategies.
Outcome:	The proposed model can compress the size of XLM-R and MiniLM by more than 50% while the performance is only reduced by about 1%.

Copied to clipboard

Jiaxin Wen, Yeshuang Zhu, Jinchao Zhang, Jie Zhou, Minlie Huang

Challenge:	Existing methods for generating counterfactuals rely on human efforts or task-specific designs.
Approach:	They propose to use a fully automatic and task-agnostic CAD generation framework to generate diverse counterfactuals.
Outcome:	The proposed framework outperforms human-in-the-loop and task-specific CAD methods on multiple out-of-domain and challenge benchmarks.

Simple and Effective Multi-Token Completion from Masked Language Models (2023.findings-eacl)

Copied to clipboard

Oren Kalinsky, Guy Kushilevitz, Alexander Libov, Yoav Goldberg

On the Effects of Transformer Size on In- and Out-of-Domain Calibration (2021.findings-emnlp)

Challenge:	Pre-trained neural masked language models are limited to predicting a single token . recent pre-tried LMs like T5 do allow predicting multi-token completions, but are more expensive to train and run.
Approach:	They propose two ways to adapt pre-trained masked language models to produce multi-token completions.
Outcome:	The proposed method surpasses current state-of-the-art models while being more parameter efficient.

Copied to clipboard

Soham Dan, Dan Roth

Collaboration or Corporate Capture? Quantifying NLP’s Reliance on Industry Artifacts and Contributions (2024.acl-long)

Challenge:	Large pre-trained transformer language models are notoriously expensive to train . prior work has developed smaller, more compact models to reduce training costs .
Approach:	They propose to develop smaller, more compact transformer language models which can be calibrated in-domain . they show that smaller models can achieve competitive calibration compared to larger models .
Outcome:	The proposed models achieve competitive calibration and better calibration than larger models on a wide range of tasks.

Copied to clipboard

Will Aitken, Mohamed Abdalla, Karen Rudie, Catherine Stinson

Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers (2021.naacl-main)

Challenge:	EMNLP 2022 citations are three times greater than expected for pre-trained models . industry participation in the Association of Computational Linguistics (ACL) anthology has increased 180% from 2017 to 2022.
Approach:	They surveyed 100 papers published at EMNLP 2022 to determine the ratio of their citations to industry models.
Outcome:	a new study shows that industry citations are three times greater than expected . the study aims to better understand whether industry collaboration is still collaboration . industry participation in the 2023 AI index report is the top takeaway .

Copied to clipboard

Andrew Silva, Pradyumna Tambwekar, Matthew Gombolay

Improving AMR Parsing with Sequence-to-Sequence Pre-training (2020.emnlp-main)

Challenge:	Existing pre-trained language models are not fully considered for societal biases . pre-training models can be useful for many NLP tasks, but they can be harmful when used at scale.
Approach:	They investigate gender and racial bias across pre-trained language models . they evaluate bias within pre-trainers using three metrics: WEAT, sequence likelihood, and pronoun ranking.
Outcome:	The proposed model fails to detect gender and racial biases in pre-trained models . the model is ineffective when word embedding, demonstrating the need for more robust bias testing in transformers.

Copied to clipboard

Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, Guodong Zhou

Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity (2022.findings-emnlp)

Challenge:	Abstract meaning representation (AMR) parsing is limited by the size of curated datasets.
Approach:	They propose a seq2seq pre-training approach to build pre-trained models on three relevant tasks.
Outcome:	The proposed model improves performance on three relevant tasks while maintaining the response of pre-trained models.

Copied to clipboard

Dennis Ulmer, Jes Frellsen, Christian Hardmeier

TANet: Thread-Aware Pretraining for Abstractive Conversational Summarization (2022.findings-naacl)

Challenge:	Using low-resource languages, we assess the quality of uncertainty estimates from a wide array of approaches, but with more data.
Approach:	They train models on sub-sampled datasets in three different languages to assess the confidence of a neural classifier.
Outcome:	The proposed models train on sub-sampled datasets in three different languages and show that the quality of uncertainty estimates suffers with more data.

Copied to clipboard

Ze Yang, Christian Wang, Zhoujin Tian, Wei Wu, Zhoujun Li

Multi-stage Pre-training over Simplified Multimodal Pre-training Models (2021.acl-long)

Challenge:	Existing pre-trained language models are difficult to apply to abstractive conversational summarization tasks.
Approach:	They propose a thread-aware Transformer-based network that incorporates contextual dependency into the conversational summarization model.
Outcome:	The proposed model can be applied to real conversations using a large-scale pretraining dataset.

Copied to clipboard

Tongtong Liu, Fangxiang Feng, Xiaojie Wang

Structure-Aware Pre-Training for Table-to-Text Generation (2021.findings-acl)

Challenge:	Existing multimodal pre-training models require large amounts of training data and have huge model sizes, making them impossible to apply in low-resource situations.
Approach:	They propose a multi-stage pre-training method which uses information at different granularities from word, phrase to sentence in both texts and images to pre-train a model in stages.
Outcome:	The proposed method outperforms the original model in Image-Text Retrieval task and outperformed the original LXMERT model in downstream tasks.

Copied to clipboard

Xinyu Xing, Xiaojun Wan

Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders (2021.acl-long)

Challenge:	Pretraining techniques have achieved great success on table-to-text generation.
Approach:	They propose a pre-trained model that is trained with tables and their contexts to generate fluent text from table input.
Outcome:	The proposed model can understand the structured input table and generate fluent text.

Copied to clipboard

Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju, Tong Xiao, Jingbo Zhu

Adaptive Textual Label Noise Learning based on Pre-trained Models (2023.findings-emnlp)

Challenge:	End-to-end Speech Translation (E2E ST) encoders lack global context representation, whereas MT encoder lacks it.
Approach:	They propose a Stacked Acoustic-and-Textual Encoding method for speech translation . they propose an adaptor module to alleviate representation inconsistency .
Outcome:	The proposed method achieves state-of-the-art BLEU scores of 18.3 and 25.2 on two ST tasks.

Copied to clipboard

Shaohuan Cheng, Wenyu Chen, Fu Mingsheng, Xuanting Xie, Hong Qu

Integrating Multimodal Information in Large Pretrained Transformers (2020.acl-main)

Challenge:	Existing approaches to learning with noisy labels are limited due to the time and labor costs involved.
Approach:	They propose an adaptive warm-up and hybrid training frameworks to learn with noisy labels based on pre-trained models.
Outcome:	The proposed approach performs comparable or even surpasses state-of-the-art methods in various noise scenarios, including scenarios with the mixture of multiple types of noise.

Copied to clipboard

Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, Ehsan Hoque

JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs (2021.findings-acl)

Challenge:	Recent Transformer-based contextual word representations have shown state-of-the-art performance in multiple disciplines within NLP.
Approach:	They propose an attachment to BERT and XLNet that allows them to accept multimodal nonverbal data during fine-tuning.
Outcome:	The proposed attachment allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning.

Copied to clipboard

Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, Minlie Huang

A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation (2022.naacl-main)

Challenge:	Existing pre-trained models for knowledgegraph-to-text generation ignore graph structure during encoding and lack elaborate pre-training tasks to explicitly model graph-text alignments.
Approach:	They propose a graph-text joint representation learning model called JointGT which incorporates a structure-aware semantic aggregation module into each Transformer layer to preserve the graph structure.
Outcome:	The proposed model achieves state-of-the-art performance on various KG-to-text datasets.

Copied to clipboard

David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, Sam Manthalu

Bridging Textual and Tabular Worlds for Fact Verification: A Lightweight, Attention-Based Model (2024.lrec-main)

Challenge:	Low-resource languages are left out of large-scale pretraining datasets . authors explore how to leverage existing pre-trained models to create low-resourced translation systems for 16 African languages.
Approach:	They investigate how large-scale pre-trained models can be used to create low-resource translation systems for 16 African languages.
Outcome:	The proposed models can translate between hundreds of languages even though there is little parallel data available for training.

Copied to clipboard

Shirin Dabbaghi Varnosfaderani, Canasai Kruengkrai, Ramin Yahyapour, Junichi Yamagishi

Transferable Post-training via Inverse Value Learning (2025.naacl-long)

Challenge:	Existing fact-checking systems rely on extensive preprocessing and rule-based transformations, leading to potential context loss or misleading encodings.
Approach:	They propose a simple yet powerful model that nullifies the need for modality conversion, thereby preserving the original evidence’s context.
Outcome:	The proposed model nullifies the need for modality conversion, preserving the original evidence’s context.

Copied to clipboard

Xinyu Lu, Xueru Wen, Yaojie Lu, Bowen Yu, Hongyu Lin, Haiyang Yu, Le Sun, Xianpei Han, Yongbin Li

On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers (2020.findings-emnlp)

Challenge:	Existing algorithms for post-training large datasets are requiring a large computational effort.
Approach:	They propose to model the changes at logits level during post-training using a separate neural network . they demonstrate that the value network can be seamlessly integrated with another pre-trained model .
Outcome:	The proposed model can be integrated with another pre-trained model during inference, enabling similar capability enhancements.

Copied to clipboard

Marius Mosbach, Anna Khokhlova, Michael A. Hedderich, Dietrich Klakow

LeeBERT: Learned Early Exit for BERT with cross-level optimization (2021.acl-long)

Challenge:	linguistic knowledge encoded in pre-trained contextual embeddings is poorly understood . fine-tuning can be used to investigate the representations of pre-train models .
Approach:	They propose to investigate fine-tuning of contextualized embedding models through sentence-level probing.
Outcome:	The proposed method improves probing accuracy for three pre-trained models.

Copied to clipboard

Wei Zhu

MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark (2021.eacl-main)

Challenge:	Pre-trained language models are resource exhaustive and computationally expensive for industrial scenarios.
Approach:	They propose a learning scheme to learn from each other to speed up inference . they ask each exit to learn the weights of different loss terms, instead of learning only from the last layer .
Outcome:	The proposed scheme improves state-of-the-art (SOTA) early exit methods for pre-trained models on the GLUE benchmark.

Copied to clipboard

Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, Yashar Mehdad

CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations (2022.coling-1)

Challenge:	Existing datasets for task-oriented dialog systems are limited and expensive . current models are based on the simple intent and slot detection paradigm for non-compositional queries.
Approach:	They propose to use a multilingual dataset to scale semantic parsing models to new languages . they demonstrate an average improvement of +6.3 points on Slot F1 for existing datasets .
Outcome:	The proposed model achieves an average improvement of +6.3 points on Slot F1 over existing models.

Copied to clipboard

Borun Chen, Hongyin Tang, Jiahao Bu, Kai Zhang, Jingang Wang, Qifan Wang, Hai-Tao Zheng, Wei Wu, Liqian Yu

Data Annealing for Informal Language Understanding Tasks (2020.findings-emnlp)

Challenge:	Pre-trained language models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding.
Approach:	They propose a Chinese pre-trained language model that implicitly encodes words into characters . they propose 'contrastive learning over word' and 'character' representations to improve learning .
Outcome:	The proposed model can encode words into fine-grained representations without modification of production pipelines.

Copied to clipboard

Jing Gu, Zhou Yu

CogBERT: Cognition-Guided Pre-trained Language Models (2022.coling-1)

Challenge:	Existing models that improve formal and informal language understanding tasks do not transfer to informal data directly.
Approach:	They propose a data annealing transfer learning procedure to bridge the performance gap on informal natural language understanding tasks.
Outcome:	The proposed procedure outperforms state-of-the-art models on three common tasks.

Copied to clipboard

Xiao Ding, Bowen Chen, Li Du, Bing Qin, Ting Liu

MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension (2024.emnlp-main)

Challenge:	Existing methods fine-tune pre-trained models on cognitive data, ignoring the semantic gap between texts and cognitive signals.
Approach:	They propose a framework that can induce fine-grained cognitive features from cognitive data and incorporate them into pre-trained language models by adaptively adjusting the weight of cognitive features for different NLP tasks.
Outcome:	The proposed framework can induce fine-grained cognitive features from cognitive data and incorporate them into BERT by adaptively adjusting weight of cognitive features for different NLP tasks.

Copied to clipboard

Ting Liu, Zunnan Xu, Yue Hu, Liangtao Shi, Zhiqiang Wang, Quanjun Yin

Green & Sustainable NLP

Challenge:	Existing methods for Referring Expression Comprehension (REC) lack specific domain abilities for precise local visual perception and visual-language alignment.
Approach:	They propose a framework for Parameter-Efficient Transfer Learning to localize a visual region via natural language using a prior-guided prior.
Outcome:	The proposed framework achieves the best accuracy compared to the current methods with only 1.41% tunable backbone parameters.

Momentum Contrastive Pre-training for Question Answering (2022.emnlp-main)

Copied to clipboard

Minda Hu, Muzhi Li, Yasheng Wang, Irwin King

Natural Language Interfaces

Challenge:	Existing methods for extractive Question Answering generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching.
Approach:	They propose a method to align the answer probability between cloze-like and natural query-passage sample pairs.
Outcome:	The proposed method improves on three benchmarking QA datasets on supervised and zero-shot scenarios.

CM-Off-Meme: Code-Mixed Hindi-English Offensive Meme Detection with Multi-Task Learning by Leveraging Contextual Knowledge (2024.lrec-main)

Copied to clipboard

Gitanjali Kumari, Dibyanayan Bandyopadhyay, Asif Ekbal, Vinutha B. NarayanaMurthy

Efficient Constituency Parsing by Pointing (2020.acl-main)

Challenge:	Existing studies on detecting offensive memes have focused on identifying them as implicit and explicit . detecting memes requires contextual knowledge, but there is no such dataset for the code-mixed Hindi-English domain.
Approach:	They propose an end-to-end multitask model that integrates contextual knowledge and psycho-linguistic knowledge to detect offensive memes.
Outcome:	The proposed model is able to detect offensive memes and explicit memes in a large-scale dataset.

Copied to clipboard

Thanh-Tung Nguyen, Xuan-Phi Nguyen, Shafiq Joty, Xiaoli Li

MMNMT: Modularizing Multilingual Neural Machine Translation with Flexibly Assembled MoE and Dense Blocks (2023.emnlp-main)

Challenge:	Constituency parsing is a core task in natural language processing (NLP) Existing methods for constituency paring are greedy transition-based or globally optimized.
Approach:	They propose a constituency parsing model that casts the problem into a series of pointing tasks.
Outcome:	The proposed model achieves 92.78 F1 without pre-trained models, which is faster than existing models.

Copied to clipboard

Shangjie Li, Xiangpeng Wei, Shaolin Zhu, Jun Xie, Baosong Yang, Deyi Xiong

ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing (2023.emnlp-main)

Challenge:	Mixture-of-Experts (MoE) based sparse architectures are prone to overfitting on low-resource language translation.
Approach:	They propose a modularized MNMT framework that flexibly assembles dense and MoE-based sparse modules to achieve the best of both worlds.
Outcome:	The proposed framework outperforms existing models on low-resource language translation and zero-shot translation on benchmark datasets.

Copied to clipboard

Nam Nguyen, Thang Phan, Duc-Vu Nguyen, Kiet Nguyen

Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents (2021.naacl-main)

Challenge:	English and Chinese have seen the strong development of transformer-based language models for natural language processing tasks.
Approach:	They present a monolingual pre-trained language model for Vietnamese social media texts . they explore emotion recognition, hate speech detection, sentiment analysis, spam reviews detection .
Outcome:	The proposed model outperforms the existing models on Vietnamese social media tasks with fewer parameters.

Copied to clipboard

Mohammad Kachuee, Hao Yuan, Young-Bum Kim, Sungjin Lee

ViNLI: A Vietnamese Corpus for Studies on Open-Domain Natural Language Inference (2022.coling-1)

Challenge:	End-to-end deep learning methods that focus on user satisfaction are challenging due to the required annotation costs and turnaround times.
Approach:	They propose a self-supervised contrastive learning approach that leverages the pool of unlabeled data to learn user-agent interactions.
Outcome:	The proposed approach reduces the required number of annotations while improving generalization on unseen skills.

Copied to clipboard

Tin Van Huynh, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen

Textual Inference

Challenge:	a large-scale corpus is needed for studies on natural language inference (NLI) for Vietnamese, which can be considered a low-resource language.
Approach:	They propose a corpus for evaluating Vietnamese natural language inference models . they use a human-annotated corpus extracted from more than 800 online news articles .
Outcome:	The ViNLI corpus is created and evaluated with a strict process of quality control . the best system performance is still far from human performance (a 14.20% gap in accuracy).

Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis (2023.emnlp-main)

Copied to clipboard

Seraphina Goldfarb-Tarrant, Björn Ross, Adam Lopez

Vision-aided Unsupervised Constituency Parsing with Multi-MLLM Debating (2025.findings-acl)

Challenge:	Existing evidence of demographic bias in SA systems is limited to a handful of languages, and it is costly to create supervised training data in a new language.
Approach:	They use counterfactual evaluation to test whether gender or racial biases are imported when using cross-lingual transfer . r&r is much more prevalent than gender biase .
Outcome:	The proposed model is compared with monolingual systems in five languages and shows that it is biased more than monolingual ones.

Copied to clipboard

Dong Zhang, Haiyan Tian, Qingying Sun, Shoushan Li

Investigating the Robustness of Natural Language Generation from Logical Forms via Counterfactual Samples (2022.emnlp-main)

Challenge:	Existing approaches require explicit cross-modal alignment, but new approaches address these challenges.
Approach:	They propose a framework for vision-aided unsupervised constituency parsing . they leverage multimodal large language models pre-trained on diverse image-text or video-text data .
Outcome:	The proposed framework achieves state-of-the-art performance on image-text and video-text datasets, improving robustness and accuracy.

Copied to clipboard

Chengyuan Liu, Leilei Gan, Kun Kuang, Fei Wu

Robustness in NLP

Challenge:	State-of-the-art methods based on pre-trained models have achieved remarkable performance on the standard test dataset.
Approach:	They propose to incorporate hierarchical structure of logical forms into the model and exploit automatically generated counterfactual data for training.
Outcome:	The proposed method is effective to alleviate spurious correlations between the headers of the tables and operators of the logical form.

CoMave: Contrastive Pre-training with Multi-scale Masking for Attribute Value Extraction (2023.findings-acl)

Copied to clipboard

Xinnan Guo, Wentao Deng, Yongrui Chen, Yang Li, Mengdi Zhou, Guilin Qi, Tianxing Wu, Dong Yang, Liubin Wang, Yong Pan

Dual-Channel Evidence Fusion for Fact Verification over Texts and Tables (2022.naacl-main)

Challenge:	Existing methods to extract product features from unstructured text still suffer from problems . e-commerce platforms are focusing on multi-scale values, which can be confusing .
Approach:	They propose a pre-training technique to automatically obtain attribute value pairs from product descriptions to aid e-commerce.
Outcome:	The proposed method improves on the existing token-level masking strategy and achieves state-of-the-art on four benchmarks.

Copied to clipboard

Nan Hu, Zirui Wu, Yuxuan Lai, Xiao Liu, Yansong Feng

Fact & Claim Verification

Challenge:	Existing fact extraction and verification tasks only consider evidence of a single format . Existing models convert evidence into either sentences or tables, thus losing context information .
Approach:	They propose a Dual Channel Unified Format fact verification model which unifies various evidence into parallel streams, i.e., natural language sentences and a global evidence table, simultaneously.
Outcome:	The proposed model outperforms existing models in two formats by a large margin . it makes the most of existing tables and tables to absorb evidence of two formats .

An empirical study of validating synthetic data for formula generation (2025.findings-naacl)

Copied to clipboard

Usneek Singh, José Cambronero, Sumit Gulwani, Aditya Kanade, Anirudh Khatry, Vu Le, Mukul Singh, Gust Verbruggen

Using Social and Linguistic Information to Adapt Pretrained Representations for Political Perspective Identification (2021.findings-acl)

Challenge:	Large language models (LLMs) can be leveraged to help write formulas in spreadsheets, but formula data resources are scarce, limiting the ability to fine-tune them.
Approach:	They validate a corpus of formulas with a model to generate synthetic natural language utterances for fine-tuning.
Outcome:	The proposed model generates synthetic natural language utterances with a model that is accurate enough to fine-tune them.

Copied to clipboard

Chang Li, Dan Goldwasser

Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization (2025.findings-naacl)

Challenge:	a new framework for political perspective detection is proposed to improve text training costs . current deep learning models lack the ability to focus on text span for bias detection .
Approach:	They propose a framework that pretrains the text model using social and linguistic contexts . they demonstrate that the framework improves performance by identifying bias-related text spans based on entity mentions and news sharing .
Outcome:	The proposed framework improves on two news bias datasets and improves performance on the general source and task.

Copied to clipboard

Yen-Ju Lu, Ting-Yao Hu, Hema Swetha Koppula, Hadi Pouransari, Jen-Hao Rick Chang, Yin Xia, Xiang Kong, Qi Zhu, Xiaoming Simon Wang, Oncel Tuzel, Raviteja Vemulapalli

Natural Language Interfaces

Challenge:	Empirical results demonstrate that our method improves dialogue summarization, achieving a 1.5% increase in ROUGE scores and a 0.3% improvement in BERT scores in few-shot settings.
Approach:	They propose Mutual Reinforcing Data Synthesis (MRDS) within large language models to enhance few-shot dialogue summarization task.
Outcome:	Empirical results show that the proposed method improves dialogue summarization, achieving a 1.5% increase in ROUGE scores and a 0.3% improvement in BERT scores in few-shot settings.

Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Models (2023.acl-long)

Copied to clipboard

Myles Foley, Ambrish Rawat, Taesung Lee, Yufang Hou, Gabriele Picco, Giulio Zizzo

Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2022.naacl-main)

Challenge:	generative large language models (LLMs) are widely used but fine-tuned to improve performance on downstream applications leads to violations of model licenses, model theft, and copyright infringement.
Approach:	They propose to trace back the origin of a model trained to its pre-trained base model . they use different knowledge levels and attribution strategies to find out how the model was trained .
Outcome:	The proposed method can trace back 8 out of 10 fine tuned models with different knowledge levels and attribution strategies.

Copied to clipboard

Xiangru Tang, Alexander Fabbri, Haoran Li, Ziming Mao, Griffin Adams, Borui Wang, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev

Information Extraction & Text Mining

Challenge:	Existing pre-trained summarization models produce text that is factually inconsistent with the input.
Approach:	They present a scale-based scale for Likert rating and a scoring algorithm for Best-Worst Scaling to improve crowdsourcing reliability.
Outcome:	The proposed model is more reliable than existing models on two news summarization datasets.

Template-free Prompt Tuning for Few-shot NER (2022.naacl-main)

Copied to clipboard

Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, Xuanjing Huang

RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter (2024.findings-acl)

Challenge:	Prompt-based methods have been successfully applied in few-shot learning tasks . however, when applied to token-level labeling tasks, it would be time-consuming to enumerate the template queries over all potential entity spans.
Approach:	They propose a method to reformulate NER tasks as LM problems without templates.
Outcome:	The proposed method is 30.12 times faster than the template-based method under few-shot settings.

Copied to clipboard

Meng Cao, Haoran Tang, Jinfa Huang, Peng Jin, Can Zhang, Ruyang Liu, Long Chen, Xiaodan Liang, Li Yuan, Ge Li

Green & Sustainable NLP

Challenge:	Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries.
Approach:	They propose to conduct efficient text-video Retrieval with a salient-and-correlated AdaPter . they propose a low-rank modulation module to refine per-image features from frozen CLIP backbone .
Outcome:	Experiments on four TVR datasets show that the proposed method performs better than other methods.

RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question (2023.findings-acl)

Copied to clipboard

Alireza Mohammadshahi, Thomas Scialom, Majid Yazdani, Pouya Yanki, Angela Fan, James Henderson, Marzieh Saeidi

Question Answering

Challenge:	Existing metrics for evaluating the quality of automatically generated questions are expensive and penalise valid questions that may not have high lexical or semantic similarity to the reference questions.
Approach:	They propose a question-answering and span scorer metric based on the answerability of the candidate question given the context.
Outcome:	The proposed metric has higher correlation with human judgment without relying on the reference question.

Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment (2023.acl-long)

Copied to clipboard

Tianshu Yu, Haoyu Gao, Ting-En Lin, Min Yang, Yuchuan Wu, Wentao Ma, Chao Wang, Fei Huang, Yongbin Li

EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering (2020.emnlp-main)

Challenge:	Existing speech-text pre-training methods are limited to one or two specific tasks, despite their success in speech-language processing tasks.
Approach:	They propose a temporal position prediction task to capture the speech-text alignment . they use a textual dialog pre-training task to generalize a response selection task .
Outcome:	The proposed model is superior in learning speech-text alignment and multi-turn dialog context.

Copied to clipboard

Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, Preslav Nakov

Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models (2022.coling-1)

Challenge:	EXAMS is a benchmark dataset for cross-lingual and multilingual question answering for high school examinations.
Approach:	They propose to use EXAMS to evaluate cross-lingual and multilingual question answering for high school examinations.
Outcome:	The proposed model can be used to explore multilingual reasoning and knowledge transfer methods and pre-trained models in schools in different languages, which was not possible by now.

Copied to clipboard

Zichun Yu, Tianyu Gao, Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Maosong Sun, Jie Zhou

Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference (2022.naacl-main)

Challenge:	Prompting has shown to be sample efficient compared to fine-tuning with pre-trained models.
Approach:	They propose a fully automatic prompting method that uses natural language prompts on sequence-to-sequence models and a beam search method to generate a large amount of label sequence candidates.
Outcome:	The proposed method significantly outperforms other no-manual-design methods on single label words and generates large amount of label sequence candidates.

Copied to clipboard

Emīls Kadiķis, Vaibhav Srivastav, Roman Klinger

Reasoning

Challenge:	a method for learning an NLI model is time-consuming and resource-intensive, but it can save time and resources.
Approach:	They propose a method for predicting model performance without fine-tuning it . they compare sentence embeddings with cosine similarity to classifiers .
Outcome:	The proposed method can save time and resources by comparing pre-trained models to real-world datasets.

JParaCrawl: A Large Scale Web-Based English-Japanese Parallel Corpus (2020.lrec-1)

Copied to clipboard

Makoto Morishita, Jun Suzuki, Masaaki Nagata

Machine Translation

Challenge:	Recent machine translation algorithms rely on parallel corpora, but only some resource-rich language pairs can benefit from them.
Approach:	They construct a parallel corpus for English-Japanese, which has 8.7 million sentence pairs . they use a web crawler to automatically align parallel sentences in the corpus .
Outcome:	The proposed corpus includes a broader range of domains and can be trained with a pre-trained model.

DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models (2023.acl-long)

Copied to clipboard

Xuxi Chen, Tianlong Chen, Weizhu Chen, Ahmed Hassan Awadallah, Zhangyang Wang, Yu Cheng

An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese (P19-1)

Challenge:	Conventional fine-tuning works through updating all of the parameters in the pre-trained model, but as the size of pre-train models grows, it can be time-consuming and computationally expensive.
Approach:	They propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights.
Outcome:	The proposed framework saves 25% inference FLOPs while maintaining competitive downstream performance.

Copied to clipboard

Enkhbold Bataa, Joshua Wu

Information Retrieval

Challenge:	Text-based transfer learning techniques can be used to perform downstream tasks.
Approach:	They propose to use text-based transfer learning techniques to pre-train a language model in an unsupervised manner and leverage them to perform effective on downstream tasks.
Outcome:	The proposed model performs better than task-specific models trained on 3 times as much data and is as effective for language modeling pre-trained on 1/30 of the data.

Multilingual Models for ASR in Chibchan Languages (2024.naacl-long)

Copied to clipboard

Rolando Coto-Solano, Tai Wan Kim, Alexander Jones, Sharid Loáiciga

Low-Rank Updates of pre-trained Weights for Multi-Task Learning (2023.findings-acl)

Challenge:	Existing algorithms for low resource-intensive languages are not available for these languages . a paper comparing the performance of different models and algorithms for these extremely low resource languages is presented.
Approach:	They propose to fine-tune four ASR algorithms to create monolingual models for Bribri and Cabécar . they then use the best performing algorithm to train joint and transfer learning models for both languages .
Outcome:	The proposed algorithms are effective in both Bribri and Cabécar, but especially in Bribri.

Copied to clipboard

Alexandre Audibert, Massih R Amini, Konstantin Usevich, Marianne Clausel

Unsupervised Fine-tuning for Text Clustering (2020.coling-main)

Challenge:	Multi-task learning is a popular approach for learning with pre-trained models due to the complexity of the tasks and the challenges associated with fine-tuning large pre-train models.
Approach:	They propose a new approach for Multi-task learning which is based on stacking the weights of Neural Networks as a tensor.
Outcome:	The proposed approach achieves equivalent performance to the state-of-the-art on the general language understanding evaluation benchmark by training only 0.3 of the parameters per task while not modifying the baseline weights.

Copied to clipboard

Shaohan Huang, Furu Wei, Lei Cui, Xingxing Zhang, Ming Zhou

Low-Resource NLP

Challenge:	Existing approaches to text clustering fine-tune pre-trained models have been limited.
Approach:	They propose a method to fine-tune pre-trained models unsupervisedly for text clustering by learning text representations and cluster assignments using a clustering oriented loss.
Outcome:	The proposed model outperforms baseline methods and achieves state-of-the-art results on three text clustering datasets.

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022.emnlp-main)

Copied to clipboard

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, Luo Si

Visual Data in NLP

Challenge:	Existing pre-trained vision-language models suffer from inefficiency and linguistic signal overwhelmed by long visual sequences in cross-modal alignment.
Approach:	They propose a vision-language foundation model with cross-modal skip-connections that can be pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives.
Outcome:	The proposed model achieves state-of-the-art results on a wide range of vision-language downstream tasks, including image captioning, image-text retrieval, visual grounding and visual question answering.

BERTGen: Multi-task Generation through BERT (2021.acl-long)

Copied to clipboard

Faidon Mitzalis, Ozan Caglayan, Pranava Madhyastha, Lucia Specia

Entity Enhanced BERT Pre-training for Chinese NER (2020.emnlp-main)

Challenge:	Recent work in unsupervised and self-supervised pre-training has revolutionised the field of natural language understanding (NLU).
Approach:	They propose to use multimodal and multilingual pre-trained models to extend BERT by fusing them together for language generation tasks.
Outcome:	The proposed model outperforms baseline models in image captioning, machine translation and multimodal machine translation tasks and is competitive with supervised counterparts.

Copied to clipboard

Chen Jia, Yuefeng Shi, Qinrong Yang, Yue Zhang

Simplify the Usage of Lexicon in Chinese NER (2020.acl-main)

Challenge:	Character-level BERT pre-trained in Chinese suffers from lacking lexicon information, which shows effectiveness for Chinese NER.
Approach:	They propose a semi-supervised method to integrate lexicon into pre-trained LMs in Chinese . they extract an entity lexiconal from raw text and integrate it into BERT .
Outcome:	The proposed method is highly effective and achieves the best results on a news dataset and two datasets annotated by the authors.

Copied to clipboard

Ruotian Ma, Minlong Peng, Qi Zhang, Zhongyu Wei, Xuanjing Huang

BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation (2021.emnlp-main)

Challenge:	Named entity recognition (NER) is concerned with the identification of named entities in unstructured text.
Approach:	They propose a method for incorporating word lexicon into character representations . experimental results show method can be easily incorporated with pre-trained models .
Outcome:	The proposed method achieves 6.15 times faster inference speed and better performance on four benchmark Chinese NER datasets.

Copied to clipboard

Haoran Xu, Benjamin Van Durme, Kenton Murray

VQA-Augmented Machine Translation with Cross-Modal Contrastive Learning (2025.findings-emnlp)

Challenge:	Existing methods for incorporating pre-trained models into NMT systems are non-trivial and lack a comparison of the impact that other pre-trainers may have on translation performance.
Approach:	They propose to use the input of a bilingual pre-trained language model as the input for NMT encoders and a stochastic layer selection approach to ensure sufficient utilization of contextualized embeddings.
Outcome:	The proposed bilingual pre-trained language model outperforms all other pre-train models on the IWSLT’14 dataset and the proposed dual-directional translation model.

Copied to clipboard

Zhihui Zhang, Shiliang Sun, Jing Zhao, Tengfei Song, Hao Yang

Towards Semi-Supervised Learning for Deep Semantic Role Labeling (D18-1)

Challenge:	Existing multimodal machine translation methods often extract visual features using pre-trained models while learning text features from scratch, leading to representation imbalance.
Approach:	They propose a cross-modal VQA-augmented multimodal machine translation method . it aligns image-source text pairs and image-question text pairs through dual-text contrastive learning .
Outcome:	The proposed method outperforms state-of-the-art methods on multiple evaluation metrics.

Copied to clipboard

Sanket Vaibhav Mehta, Jay Yoon Lee, Jaime Carbonell

Learning Word Vectors for 157 Languages (L18-1)

Challenge:	Existing methods for semantic role labeling require an immense amount of semantic-role corpora and are therefore not suitable for low-resource languages or domains.
Approach:	They propose a semi-supervised method that outperforms the state-of-the-art on SRL . method explicitly enforcs syntactic constraints by augmenting the training objective with a syntastic-inconsistency loss component.
Outcome:	The proposed method outperforms the state-of-the-art on limited SRL training corpora on CoNLL-2012 English section.

Copied to clipboard

Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, Tomas Mikolov

ReadOnce Transformers: Reusable Representations of Text for Transformers (2021.acl-long)

Challenge:	Distributed word representations, or word vectors, have been used in natural language processing for many tasks.
Approach:	They propose to use the encyclopedia Wikipedia and the common crawl corpus to train distributed word representations on large corpora and use them in downstream tasks.
Outcome:	The proposed model performs very well on 10 languages for which evaluation dataset exists.

Copied to clipboard

Shih-Ting Lin, Ashish Sabharwal, Tushar Khot

Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings (2022.acl-long)

Challenge:	ReadOnce Transformers is a task-independent, task-dependent, and compressed representation of text.
Approach:	They propose a transformer-based model that can build an information-capturing, task-independent, and compressed representation of text.
Outcome:	The proposed model can build an information-capturing, task-independent, and compressed representation of text.

Copied to clipboard

Jiangbin Zheng, Yile Wang, Ge Wang, Jun Xia, Yufei Huang, Guojiang Zhao, Yue Zhang, Stan Li

CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training (2023.findings-acl)

Challenge:	Contextualized embeddings are expensive and resource-demanding, hence environmentally unfriendly.
Approach:	They propose a method to convert contextualized embeddings from pre-trained models into static embeddables using synonym knowledge and weighted vector distribution.
Outcome:	The proposed method outperforms baseline embeddings by a large margin through extrinsic and intrinsic tasks.

Copied to clipboard

Linhao Dong, Zhecheng An, Peihao Wu, Jun Zhang, Lu Lu, Ma Zejun

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models (2023.emnlp-main)

Challenge:	Speech-to-text training and language model distillation are used to bridge the representations between speech and text.
Approach:	They propose a pre-training paradigm that integrates speech and text into a single frame-to-token alignment.
Outcome:	The proposed paradigm outperforms the state-of-the-art model on intent classification and slot filling tasks.

Copied to clipboard

Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, Dongfang Liu

SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge (2020.emnlp-main)

Challenge:	Existing prompt tuning methods only introduce prompts at the input layer, limiting performance and leaving large room for improvement.
Approach:	They propose a method that involves tuning a small set of soft prompts for pre-trained language models.
Outcome:	The proposed method outperforms state-of-the-art methods with pre-trained models on the SuperGLUE benchmark.

Copied to clipboard

Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, Minlie Huang

Improving Abstractive Dialogue Summarization with Speaker-Aware Supervised Contrastive Learning (2022.coling-1)

Challenge:	Existing pre-trained models neglect to consider linguistic knowledge of texts . existing models neglect linguistic information, which is important for sentiment analysis .
Approach:	They propose a model that introduces word-level linguistic knowledge into pre-trained models to enhance sentiment analysis by querying SentiWordNet to acquire sentiment polarity.
Outcome:	The proposed model obtains state-of-the-art performance on a variety of sentiment analysis tasks.

Copied to clipboard

Zhichao Geng, Ming Zhong, Zhangyue Yin, Xipeng Qiu, Xuanjing Huang

Enhancing Pre-trained Models with Text Structure Knowledge for Question Generation (2022.coling-1)

Challenge:	Existing summarization systems based on pre-trained models cannot recognize the unique format of the speaker-utterance pair well in the dialogue.
Approach:	They propose three speaker-aware supervised contrastive learning tasks to solve the speaker identification problem in dialogue summarization task.
Outcome:	The proposed methods improve on two mainstream dialogue summarization datasets.

Copied to clipboard

Zichen Wu, Xin Jia, Fanyi Qu, Yunfang Wu

SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities (2022.acl-long)

Challenge:	Existing question generation models treat input passage as a sequence-to-sequence generative task, but they are not aware of text structure.
Approach:	They propose to model text structure as answer position and syntactic dependency and propose a mask attention mechanism to make syntaktic structure of input passage accessible.
Outcome:	The proposed model outperforms the strong pre-trained model ProphetNet on a SQuAD dataset and achieves competitive results with the state-of-the-art model.

Copied to clipboard

Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-wen Yang, Shuyan Dong, Andy Liu, Cheng-I Lai, Jiatong Shi, Xuankai Chang, Phil Hall, Hsuan-Jui Chen, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, Hung-yi Lee

Speech & Audio in NLP

Challenge:	Existing evaluation methods for transfer learning are limited in speech research . authors show that pre-trained models transfer well across multiple tasks .
Approach:	They propose a benchmark to evaluate pre-trained models by increasing task diversity and difficulty over SUPERB.
Outcome:	The proposed benchmark increases task diversity and difficulty over SUPERB-SG.

Is Graph Structure Necessary for Multi-hop Question Answering? (2020.emnlp-main)

Copied to clipboard

Nan Shao, Yiming Cui, Ting Liu, Shijin Wang, Guoping Hu

Structured Data in NLP

Challenge:	Existing studies focus on multi-hop question answering across multiple documents or paragraphs.
Approach:	They propose a graph neural network to deal with graph structure in textual multi-hop reasoning . they propose 'self-attention' and propose removing entire graph structure may not hurt the final results .
Outcome:	The proposed model shows that graph-attention or the entire graph structure can be replaced by self-attention . hotpotQA is a widely used benchmark for multi-hop question answering .

Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification (2023.findings-acl)

Copied to clipboard

Renliang Sun, Wei Xu, Xiaojun Wan

Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models (2023.findings-emnlp)

Challenge:	Existing strategies to teach pre-trained models to generate simple texts are inadequate.
Approach:	They propose a continued pre-training strategy to teach pre-trained models to generate simple texts by randomly masking text spans in ordinary texts.
Outcome:	The proposed strategy improves on lexical simplification, sentence simplification and document-level simplification tasks over existing models.

Copied to clipboard

Raymond Li, Gabriel Murray, Giuseppe Carenini

POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training (2020.emnlp-main)

Challenge:	In recent years, pre-trained language models have become the de facto instrument for the field of natural language processing (NLP).
Approach:	They propose a method that injects linguistic structures into pre-trained language models in the parameter-efficient fine-tuning setting.
Outcome:	The proposed approach outperforms state-of-the-art methods with a comparable number of parameters.

Copied to clipboard

Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, Bill Dolan

Efficient Transformer Parameter Reuse via Zero-Token Mechanism (2026.findings-acl)

Challenge:	Existing pre-trained language models cannot be directly employed to generate text under specified lexical constraints.
Approach:	They propose a method for insertion-based text generation that inserts tokens between existing tokens in a parallel manner.
Outcome:	The proposed method is intuitive and interpretable on Wikipedia and Yelp datasets.

Copied to clipboard

Guanghao Li, Wenhao Jiang, Li Shen, Ming Tang, Chun Yuan

FreeChunker: A Cross-Granularity Chunking Framework (2026.findings-acl)

Challenge:	Existing approaches to scaling up parameter counts are impractical for users with limited computational resources.
Approach:	They propose a decoupled parameter cycling strategy that employs a head-tail decoupling strategy to decouple the first (head) and last (tail) layers from the parameter cycling process.
Outcome:	The proposed approach achieves superior performance under strict parameter constraints and significantly reduces computational overhead via early exits.

Copied to clipboard

Zhang Wenxuan, Yuan-Hao Jiang, Yang Cao, Yonghe Wu

Syntactic Text Processing

Challenge:	Existing chunking paradigms rely on static boundary identification, limiting performance . Existing methods rely only on static knowledge, resulting in hallucinated content .
Approach:	They propose a Cross-Granularity Encoding Framework that treats sentences as atomic units and shifts from static chunk segmentation to flexible retrieval supporting arbitrary sentence combinations.
Outcome:	The proposed framework avoids the computational overhead required for semantic boundary detection and enhances adaptability to complex queries.

TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models (2023.findings-emnlp)

Copied to clipboard

Amrit Nagarajan, Anand Raghunathan

Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition (2024.findings-acl)

Challenge:	Pre-training of Language Models (LMs) is a challenge due to its huge computational footprint.
Approach:	They propose a framework that improves the efficiency and accuracy of LM fine-tuning by removing padding tokens from sequences that are variable-length .
Outcome:	The proposed framework accelerates fine-tuning on diverse downstream tasks by 10.61X while producing models that are up to 1.17% more accurate compared to conventional fine-uning.

Copied to clipboard

Yukiya Hono, Koh Mitsuda, Tianyu Zhao, Kentaro Mitsui, Toshiaki Wakatsuki, Kei Sawada

Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification (2024.emnlp-main)

Challenge:	Mainstream of automatic speech recognition (ASR) has shifted from pipeline methods to end-to-end (E2E) methods.
Approach:	They propose to integrate a pre-trained speech representation model and a large language model (LLM) for automatic speech recognition in an end-to-end manner.
Outcome:	The proposed model achieves comparable performance to modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach.

Copied to clipboard

Ming Li, Jike Zhong, Chenxin Li, Liuzhuozheng Li, Nie Lin, Masashi Sugiyama

BERTifying Sinhala - A Comprehensive Analysis of Pre-trained Language Models for Sinhala Text Classification (2022.lrec-1)

Challenge:	Recent advances in fine-tuning Vision-Language Models have seen the success of prompt tuning and adapter tuning.
Approach:	They propose a method to fine-tune CLIP without introducing any overhead of extra parameters.
Outcome:	The proposed method improves CLIP by 7.27% average harmonic mean accuracy.

Copied to clipboard

Vinura Dhananjaya, Piyumal Demotte, Surangika Ranathunga, Sanath Jayasena

KnowVrDU: A Unified Knowledge-aware Prompt-Tuning Framework for Visually-rich Document Understanding (2024.lrec-main)

Challenge:	Large-scale monolingual pre-trained language models have shown promising results for high-resource as well as lowresource languages, especially for text classification.
Approach:	They provide a set of recommendations for using pre-trained models for Sinhala text classification and introduce new annotated datasets useful for future research.
Outcome:	The proposed models are far superior to existing models for Sinhala and set a strong baseline for text classification when fine-tuned.

Copied to clipboard

Yunqi Zhang, Yubo Chen, Jingzhe Zhu, Jinyu Xu, Shuai Yang, Zhaoliang Wu, Liang Huang, Yongfeng Huang, Shuai Chen

CED: Comparing Embedding Differences for Detecting Out-of-Distribution and Hallucinated Text (2024.findings-emnlp)

Challenge:	Existing methods for integrating layout and image features into pre-training language models are not suitable for few-shot settings.
Approach:	They propose to reformulate VrDU tasks into a single question-answering format with task-specific prompts and train the pre-trained model with the parameter-efficient prompt tuning method.
Outcome:	The proposed framework can be used in few-shot settings and reduces data requirements.

Copied to clipboard

Hakyung Lee, Keon-Hee Park, Hoyoon Byun, Jeyoon Yeom, Jihee Kim, Gyeong-Moon Park, Kyungwoo Song

Vector-Quantized Prompt Learning for Paraphrase Generation (2023.findings-emnlp)

Challenge:	Existing methods for detecting out-of-distribution (OOD) samples are limited due to their domain shift and computational limitations.
Approach:	They propose a training-free method to detect out-of-distribution (OOD) samples . they theoretically validate that specific auxiliary and oracle samples improve this distinction .
Outcome:	The proposed method improves the ability of pre-trained models to distinguish between ID and OOD samples in text classification and hallucination detection tasks.

Copied to clipboard

Haotian Luo, Yixin Liu, Peidong Liu, Xianggen Liu

LAMBDA: Large Language Model-Based Data Augmentation for Multi-Modal Machine Translation (2024.findings-emnlp)

Challenge:	Existing methods for paraphrase generation are difficult to understand and generate.
Approach:	They propose to generate diverse paraphrases by using instance-dependent prompts to control the generation of pre-trained models.
Outcome:	The proposed method achieves state-of-the-art on three benchmark datasets, including Quora, Wikianswers, and MSCOCO.

Copied to clipboard

Yusong Wang, Dongyuan Li, Jialun Shen, Yicheng Xu, Mingkun Xu, Kotaro Funakoshi, Manabu Okumura

Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (2024.emnlp-main)

Challenge:	Multi-modal machine translation methods are underperforming compared to pre-trained models due to lack of triplet training data.
Approach:	They propose a multi-modal machine translation method that integrates images and visual modality to enhance language understanding.
Outcome:	The proposed method can enrich the original samples and expand the dataset without requiring external images and text.

Copied to clipboard

Vatsal Gupta, Pranshu Pandya, Tushar Kataria, Vivek Gupta, Dan Roth

SynTOD: Augmented Response Synthesis for Robust End-to-End Task-Oriented Dialogue System (2024.lrec-main)

Challenge:	Language models display sensitivity to input perturbations, causing concerns about trust among users.
Approach:	They propose a methodology to examine how input perturbations affect language models across various scales, including pre-trained models and large language models.
Outcome:	The proposed methods enhance the model’s robustness to input perturbations and if exposure to one perturbation enhances or diminishes its performance with respect to other perturbations.

Copied to clipboard

Nguyen Quang Chieu, Quang-Minh Tran, Khac-Hoai Nam Bui

Robustness in NLP

Challenge:	Task-oriented dialogue systems focus on training multiple tasks such as language understanding, tracking states, and generating appropriate responses to help users achieve their specific goals.
Approach:	They exploit the ability of pre-trained models to provide synthesis responses for fine-tuning end-to-end TOD systems.
Outcome:	The proposed model outperforms baseline models on multiwoz datasets and is available for further exploitation.

The Corpus AIKIA: Using Ranking Annotation for Offensive Language Detection in Modern Greek (2024.lrec-main)

Copied to clipboard

Stella Markantonatou, Vivian Stamou, Christina Christodoulou, Georgia Apostolopoulou, Antonis Balas, George Ioannakis

3R: Enhancing Sentence Representation Learning via Redundant Representation Reduction (2025.emnlp-main)

Challenge:	OLD is a less-resourced language regarding OLD.
Approach:	They propose to annotate OLD in Modern Greek using the lexicon of offensive terms that originates from HurtLex.
Outcome:	The proposed corpus is based on the lexicon of offensive terms that originates from HurtLex and can be used to detect offensive language in modern Greek.

Copied to clipboard

Longxuan Ma, Xiao Wu, Yuxin Huang, Shengxiang Gao, Zhengtao Yu